Skip to content

Fix hybrid scan parquet reader incorrectly parse bloom filter - #22901

Merged
rapids-bot[bot] merged 55 commits into
NVIDIA:mainfrom
qbacpey:fix/bloom-header-hybrid
Jul 25, 2026
Merged

Fix hybrid scan parquet reader incorrectly parse bloom filter#22901
rapids-bot[bot] merged 55 commits into
NVIDIA:mainfrom
qbacpey:fix/bloom-header-hybrid

Conversation

@qbacpey

@qbacpey qbacpey commented Jun 16, 2026

Copy link
Copy Markdown
Contributor

Description

Closes #22933

This PR fixes bloom filtering in the hybrid-scan Parquet reader. The previous path treated the full bloom filter payload (header + bitset) as a raw bitset; the new path fetches the filter to host, parses the header, and copies only the bitset to device. It also refactors read_parquet and hybrid scan to share the same bloom-filter fetch path.

Checklist

  • I am familiar with the Contributing Guidelines.
  • New or existing tests cover these changes.
  • The documentation is up to date with these changes.

@copy-pr-bot

copy-pr-bot Bot commented Jun 16, 2026

Copy link
Copy Markdown

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

@github-actions github-actions Bot added libcudf Affects libcudf (C++/CUDA) code. Python Affects Python cuDF API. pylibcudf Issues specific to the pylibcudf package labels Jun 16, 2026
@GPUtester GPUtester moved this to In Progress in cuDF Python Jun 16, 2026
@qbacpey qbacpey changed the title Since source hybrid scan Bloom filter A/B comparison Single source hybrid scan Bloom filter A/B comparison Jun 16, 2026
Comment thread python/pylibcudf/tests/io/test_experimental_hybrid_scan.py Outdated
@qbacpey qbacpey changed the title Single source hybrid scan Bloom filter A/B comparison Single source hybrid scan bloom filtering A/B comparison Jun 16, 2026
@qbacpey
qbacpey force-pushed the fix/bloom-header-hybrid branch from ca7c8b0 to 5acd266 Compare June 17, 2026 07:50
@qbacpey qbacpey changed the title Single source hybrid scan bloom filtering A/B comparison [BUG] Fix the problem of hybrid scan reader would not parse Parquet Bloom filter header Jun 17, 2026
@qbacpey qbacpey added the bug Something isn't working label Jun 17, 2026
@qbacpey qbacpey changed the title [BUG] Fix the problem of hybrid scan reader would not parse Parquet Bloom filter header [BUG] Fix hybrid scan parquet reader incorrectly prunes row group with bloom filter Jun 22, 2026
@qbacpey qbacpey changed the title [BUG] Fix hybrid scan parquet reader incorrectly prunes row group with bloom filter [BUG] Fix hybrid scan parquet reader incorrectly parse bloom filter Jun 22, 2026
@qbacpey
qbacpey force-pushed the fix/bloom-header-hybrid branch 2 times, most recently from 399b73b to ccf7988 Compare June 24, 2026 14:22
@qbacpey qbacpey added the 2 - In Progress Currently a work in progress label Jun 25, 2026
Comment thread cpp/src/io/parquet/io_utils/parquet_io_utils.cpp
qbacpey added 5 commits June 26, 2026 20:08
…; remove outdated test for Bloom filters with real data

Need to rebuild all the code for A/B test
… read_parquet

Updated the test for hybrid scan bloom filtering to explicitly compare its behavior against the read_parquet method. Enhanced debug logging to provide detailed output of the bloom filter data being processed, ensuring consistency in row group retention between the two methods. Adjusted comments and structure for better readability and understanding of the test's purpose.
Refactored the hybrid scan bloom filter test to utilize parameterized fixtures for improved flexibility and clarity. This change allows for multiple test cases to be run with different input values, enhancing the robustness of the test against various scenarios. Updated the test structure and comments for better readability and understanding of the A/B comparison with the read_parquet method.
…and clarity

Updated the hybrid scan implementation to replace the previous method of fetching bloom filter data with a new function, `fetch_bloom_filters_to_device_async`, which strips headers and ensures 32-byte alignment. This change enhances the efficiency of bloom filter processing and simplifies the codebase. Additionally, updated related tests to reflect these changes and ensure consistency in behavior across different methods of bloom filter handling.
@qbacpey
qbacpey force-pushed the fix/bloom-header-hybrid branch from ccf7988 to df8fc33 Compare June 26, 2026 18:09
qbacpey added 3 commits June 26, 2026 22:24
Removed unused imports and functions related to bloom filter processing in the hybrid scan module. This cleanup enhances code clarity and reduces complexity. Updated the test suite to reflect these changes, ensuring that the functionality remains intact and consistent with previous implementations.
Modified copyright statements in multiple files to include "NVIDIA CORPORATION & AFFILIATES." Enhanced code readability by breaking long parameter descriptions into multiple lines in the `parquet_io_utils.hpp` file. Removed outdated debug logging and unnecessary comments in the `bloom_filter_reader.cu` and `parquet_io_utils.cpp` files to streamline the codebase.
…vice_async_impl`

Simplified comments and improved code clarity by removing outdated explanations regarding bloom filter header handling. The changes focus on enhancing readability while maintaining the functionality of fetching bloom filter data to the device. This aligns with previous efforts to streamline the hybrid scan implementation and improve performance.
Comment thread cpp/src/io/parquet/io_utils/parquet_io_utils.cpp Outdated
Comment thread cpp/src/io/parquet/bloom_filter_reader.cu
@mhaseeb123 mhaseeb123 removed their assignment Jul 21, 2026
qbacpey and others added 2 commits July 21, 2026 09:13
…gement

- Removed unnecessary `copy_dsts` vector and related null pointer handling.
- Simplified the logic for populating `copy_srcs` and `copy_sizes`.
- Ensured that device memory allocation is only performed when `device_base` is not null.
- Improved clarity and maintainability of the code by restructuring deferred read handling.

This change enhances the efficiency of the bloom filter fetching process in the Parquet I/O utilities.
@qbacpey
qbacpey requested a review from mhaseeb123 July 21, 2026 09:54

@mhaseeb123 mhaseeb123 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Few optional non-blocking comments but LGTM otherwise. Would recommend removing the Follow-up PR section from the description and instead creating an issue for better tracking. I didn't understand what "Edge cases for current bloom filter fetching function" means? Are there any unhandled edge cases in this PR?

@mhaseeb123 mhaseeb123 moved this from Burndown to Slip in libcudf Jul 21, 2026
Comment thread cpp/src/io/parquet/bloom_filter_reader.cu Outdated
@qbacpey

qbacpey commented Jul 22, 2026

Copy link
Copy Markdown
Contributor Author

Few optional non-blocking comments but LGTM otherwise. Would recommend removing the Follow-up PR section from the description and instead creating an issue for better tracking. I didn't understand what "Edge cases for current bloom filter fetching function" means? Are there any unhandled edge cases in this PR?

Thanks for the review.

There are no known unhandled edge cases in this PR. “Edge cases for the current bloom filter fetching function” refers to follow-up pytest coverage for scenarios that the committed Parquet fixtures do not cover well. I’ve removed the Follow-up PR section and opened #23393 to track this work instead.

@vuule vuule left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, few small suggestions.

Comment thread cpp/src/io/parquet/io_utils/parquet_io_utils.cpp
Comment thread cpp/src/io/parquet/io_utils/parquet_io_utils.cpp Outdated
qbacpey and others added 3 commits July 24, 2026 23:11
…filters_to_device_impl`. Updated warning message to include dynamic block size and refined mutex lock scope for better readability.
@qbacpey

qbacpey commented Jul 25, 2026

Copy link
Copy Markdown
Contributor Author

/merge

@rapids-bot
rapids-bot Bot merged commit 3a3a522 into NVIDIA:main Jul 25, 2026
136 checks passed
@qbacpey
qbacpey deleted the fix/bloom-header-hybrid branch July 25, 2026 11:44
@GregoryKimball GregoryKimball moved this from Slip to Landed in libcudf Aug 21, 2026
@GregoryKimball GregoryKimball removed this from libcudf Aug 21, 2026
@GregoryKimball GregoryKimball moved this from Slip to Landed in libcudf Aug 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

4 - Needs Review Waiting for reviewer to review or respond bug Something isn't working libcudf Affects libcudf (C++/CUDA) code. non-breaking Non-breaking change

Projects

Status: Landed

Development

Successfully merging this pull request may close these issues.

[BUG] Hybrid scan parquet reader incorrectly prunes row group with bloom filter

9 participants